Skip to content

fix: support non-finite floats in complex results - #574

Open
cathleeny wants to merge 1 commit into
mainfrom
fix/nested-nonfinite-floats
Open

cathleeny wants to merge 1 commit into
mainfrom
fix/nested-nonfinite-floats

Conversation

@cathleeny

Copy link
Copy Markdown
Collaborator

Summary

Fixes #573.

Valid SQL FLOAT and DOUBLE values containing NaN or infinities currently abort row iteration when nested inside ARRAY, MAP, or STRUCT results. The Go renderers serialize scalar leaves through encoding/json, which rejects non-finite floats.

Render these leaves as the JSON strings "NaN", "Infinity", and "-Infinity" in the shared formatter used by both Thrift and kernel-backed results. Keep SQL NULL as null rather than losing the distinction between special floats and missing values.

Compatibility

  • Finite-number formatting, top-level float values/types, and NULL handling are unchanged.
  • Preserve the kernel backend's existing floating-point MAP-key spellings: "NaN", "+Inf", and "-Inf". Thrift now accepts these previously rejected keys with the same spelling.
  • Nested special values that previously caused an error now return JSON strings. Applications decoding these leaves must use any or a custom decoder rather than plain float fields; this is documented in README and package docs.
  • No native kernel change, dependency update, or new connection option is needed.

Test Plan

  • New formatter and cross-backend tests failed before the fix.
  • Full default-build CGO_ENABLED=0 go test ./... -count=1 passed with live warehouse credentials removed from the test environment.
  • CGO_ENABLED=1 go test -race ./internal/jsonfmt ./internal/arrowscan ./internal/rows/arrowbased -count=1 passed.
  • CGO_ENABLED=0 go vet ./... passed.
  • make lint passed with zero issues.
  • In-memory Arrow tests cover FLOAT and DOUBLE, arrays, map keys/values, structs, nested containers, valid JSON, NULL distinction, and unchanged top-level numeric types. Existing finite-format parity and zero-allocation checks remain enabled.

No live warehouse tests or native-kernel-linked E2E tests were run. Both Go renderers are exercised directly with in-memory Arrow arrays.

This pull request and its description were written by Isaac.

Quote non-finite FLOAT and DOUBLE leaves in the shared JSON formatter instead of aborting row iteration. Preserve finite formatting, SQL NULL, top-level numeric types, and existing kernel MAP-key spellings. Add cross-backend regression coverage and document the encoding.

Signed-off-by: Cathleen Yan <58714163+cathleeny@users.noreply.github.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@vuanhphung vuanhphung added the skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test) label Nov 18, 2026
func BenchmarkAppendScalars(b *testing.B) {
// The last three take json.Marshal's fallback.
// The escaped string and []byte take json.Marshal's fallback.
values := []any{int32(42), int64(-9000), 3.14, 0.30000000000000004, float32(0.5), true, "alpha0", "café crème",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

math.NaN() is still in BenchmarkAppendScalars. It's now a fast-path append for jsonfmt but an error path for the encoding_json baseline, so the comparison is skewed. Could we drop it, or add a comment that the baseline errors on it?

if f < 0 {
return "-Inf"
}
return "+Inf"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we remove MapKey and use "NaN", "Infinity", and "-Infinity" consistently for both keys and values? Currently, a map containing positive infinity as both key and value renders as {"+Inf":"Infinity"}.

JDBC’s text rendering and Node’s object keys use Infinity/-Infinity; Python preserves native numeric keys. The +Inf/-Inf spelling comes from the kernel’s old formatting fallback, while Thrift previously errored on these keys. Using the same spelling throughout would simplify the contract; we can document the kernel key spelling change explicitly in the changelog.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-coverage Skip the coverage fan-out for this PR (no tracking issue opened in databricks-driver-test)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

NaN or Infinity inside ARRAY, MAP or STRUCT fails the whole result

2 participants